x
1
===============================
2
Understanding the status screen
3
===============================
4
5
  This document provides an overview of the status screen - plus tips for
6
  troubleshooting any warnings and red text shown in the UI. See README for
7
  the general instruction manual.
8
9
0) A note about colors
10
----------------------
11
12
The status screen and error messages use colors to keep things readable and
13
attract your attention to the most important details. For example, red almost
14
always means "consult this doc" :-)
15
16
Unfortunately, the UI will render correctly only if your terminal is using
17
traditional un*x palette (white text on black background) or something close
18
to that.
19
20
If you are using inverse video, you may want to change your settings, say:
21
22
  - For GNOME Terminal, go to Edit > Profile preferences, select the "colors"
23
    tab, and from the list of built-in schemes, choose "white on black".
24
25
  - For the MacOS X Terminal app, open a new window using the "Pro" scheme via
26
    the Shell > New Window menu (or make "Pro" your default).
27
28
Alternatively, if you really like your current colors, you can edit config.h
29
to comment out USE_COLORS, then do 'make clean all'.
30
31
I'm not aware of any other simple way to make this work without causing
32
other side effects - sorry about that.
33
34
With that out of the way, let's talk about what's actually on the screen...
35
36
1) Process timing
37
-----------------
38
39
  +----------------------------------------------------+
40
  |        run time : 0 days, 8 hrs, 32 min, 43 sec    |
41
  |   last new path : 0 days, 0 hrs, 6 min, 40 sec     |
42
  | last uniq crash : none seen yet                    |
43
  |  last uniq hang : 0 days, 1 hrs, 24 min, 32 sec    |
44
  +----------------------------------------------------+
45
46
This section is fairly self-explanatory: it tells you how long the fuzzer has
47
been running and how much time has elapsed since its most recent finds. This is
48
broken down into "paths" (a shorthand for test cases that trigger new execution
49
patterns), crashes, and hangs.
50
51
When it comes to timing: there is no hard rule, but most fuzzing jobs should be
52
expected to run for days or weeks; in fact, for a moderately complex project, the
53
first pass will probably take a day or so. Every now and then, some jobs
54
will be allowed to run for months.
55
56
There's one important thing to watch out for: if the tool is not finding new
57
paths within several minutes of starting, you're probably not invoking the
58
target binary correctly and it never gets to parse the input files we're
59
throwing at it; another possible explanations are that the default memory limit
60
(-m) is too restrictive, and the program exits after failing to allocate a
61
buffer very early on; or that the input files are patently invalid and always
62
fail a basic header check.
63
64
If there are no new paths showing up for a while, you will eventually see a big
65
red warning in this section, too :-)
66
67
2) Overall results
68
------------------
69
70
  +-----------------------+
71
  |  cycles done : 0      |
72
  |  total paths : 2095   |
73
  | uniq crashes : 0      |
74
  |   uniq hangs : 19     |
75
  +-----------------------+
76
77
The first field in this section gives you the count of queue passes done so far
78
- that is, the number of times the fuzzer went over all the interesting test
79
cases discovered so far, fuzzed them, and looped back to the very beginning.
80
Every fuzzing session should be allowed to complete at least one cycle; and
81
ideally, should run much longer than that.
82
83
As noted earlier, the first pass can take a day or longer, so sit back and
84
relax. If you want to get broader but more shallow coverage right away, try
85
the -d option - it gives you a more familiar experience by skipping the
86
deterministic fuzzing steps. It is, however, inferior to the standard mode in
87
a couple of subtle ways.
88
89
To help make the call on when to hit Ctrl-C, the cycle counter is color-coded.
90
It is shown in magenta during the first pass, progresses to yellow if new finds
91
are still being made in subsequent rounds, then blue when that ends - and
92
finally, turns green after the fuzzer hasn't been seeing any action for a
93
longer while.
94
95
The remaining fields in this part of the screen should be pretty obvious:
96
there's the number of test cases ("paths") discovered so far, and the number of
97
unique faults. The test cases, crashes, and hangs can be explored in real-time
98
by browsing the output directory, as discussed in the README.
99
100
3) Cycle progress
101
-----------------
102
103
  +-------------------------------------+
104
  |  now processing : 1296 (61.86%)     |
105
  | paths timed out : 0 (0.00%)         |
106
  +-------------------------------------+
107
108
This box tells you how far along the fuzzer is with the current queue cycle: it
109
shows the ID of the test case it is currently working on, plus the number of
110
inputs it decided to ditch because they were persistently timing out.
111
112
The "*" suffix sometimes shown in the first line means that the currently
113
processed path is not "favored" (a property discussed later on, in section 6).
114
115
If you feel that the fuzzer is progressing too slowly, see the note about the
116
-d option in section 2 of this doc.
117
118
4) Map coverage
119
---------------
120
121
  +--------------------------------------+
122
  |    map density : 10.15% / 29.07%     |
123
  | count coverage : 4.03 bits/tuple     |
124
  +--------------------------------------+
125
126
The section provides some trivia about the coverage observed by the
127
instrumentation embedded in the target binary.
128
129
The first line in the box tells you how many branch tuples we have already
130
hit, in proportion to how much the bitmap can hold. The number on the left
131
describes the current input; the one on the right is the value for the entire
132
input corpus.
133
134
Be wary of extremes:
135
136
  - Absolute numbers below 200 or so suggest one of three things: that the
137
    program is extremely simple; that it is not instrumented properly (e.g.,
138
    due to being linked against a non-instrumented copy of the target
139
    library); or that it is bailing out prematurely on your input test cases.
140
    The fuzzer will try to mark this in pink, just to make you aware.
141
142
  - Percentages over 70% may very rarely happen with very complex programs
143
    that make heavy use of template-generated code.
144
145
    Because high bitmap density makes it harder for the fuzzer to reliably
146
    discern new program states, I recommend recompiling the binary with
147
    AFL_INST_RATIO=10 or so and trying again (see env_variables.txt).
148
149
    The fuzzer will flag high percentages in red. Chances are, you will never
150
    see that unless you're fuzzing extremely hairy software (say, v8, perl,
151
    ffmpeg).
152
153
The other line deals with the variability in tuple hit counts seen in the
154
binary. In essence, if every taken branch is always taken a fixed number of
155
times for all the inputs we have tried, this will read "1.00". As we manage
156
to trigger other hit counts for every branch, the needle will start to move
157
toward "8.00" (every bit in the 8-bit map hit), but will probably never
158
reach that extreme.
159
160
Together, the values can be useful for comparing the coverage of several
161
different fuzzing jobs that rely on the same instrumented binary.
162
163
5) Stage progress
164
-----------------
165
166
  +-------------------------------------+
167
  |  now trying : interest 32/8         |
168
  | stage execs : 3996/34.4k (11.62%)   |
169
  | total execs : 27.4M                 |
170
  |  exec speed : 891.7/sec             |
171
  +-------------------------------------+
172
173
This part gives you an in-depth peek at what the fuzzer is actually doing right
174
now. It tells you about the current stage, which can be any of:
175
176
  - calibration - a pre-fuzzing stage where the execution path is examined
177
    to detect anomalies, establish baseline execution speed, and so on. Executed
178
    very briefly whenever a new find is being made.
179
180
  - trim L/S - another pre-fuzzing stage where the test case is trimmed to the
181
    shortest form that still produces the same execution path. The length (L)
182
    and stepover (S) are chosen in general relationship to file size.
183
184
  - bitflip L/S - deterministic bit flips. There are L bits toggled at any given
185
    time, walking the input file with S-bit increments. The current L/S variants
186
    are: 1/1, 2/1, 4/1, 8/8, 16/8, 32/8.
187
188
  - arith L/8 - deterministic arithmetics. The fuzzer tries to subtract or add
189
    small integers to 8-, 16-, and 32-bit values. The stepover is always 8 bits.
190
191
  - interest L/8 - deterministic value overwrite. The fuzzer has a list of known
192
    "interesting" 8-, 16-, and 32-bit values to try. The stepover is 8 bits.
193
194
  - extras - deterministic injection of dictionary terms. This can be shown as
195
    "user" or "auto", depending on whether the fuzzer is using a user-supplied
196
    dictionary (-x) or an auto-created one. You will also see "over" or "insert",
197
    depending on whether the dictionary words overwrite existing data or are
198
    inserted by offsetting the remaining data to accommodate their length.
199
200
  - havoc - a sort-of-fixed-length cycle with stacked random tweaks. The
201
    operations attempted during this stage include bit flips, overwrites with
202
    random and "interesting" integers, block deletion, block duplication, plus
203
    assorted dictionary-related operations (if a dictionary is supplied in the
204
    first place).
205
206
  - splice - a last-resort strategy that kicks in after the first full queue
207
    cycle with no new paths. It is equivalent to 'havoc', except that it first
208
    splices together two random inputs from the queue at some arbitrarily
209
    selected midpoint.
210
211
  - sync - a stage used only when -M or -S is set (see parallel_fuzzing.txt).
212
    No real fuzzing is involved, but the tool scans the output from other
213
    fuzzers and imports test cases as necessary. The first time this is done,
214
    it may take several minutes or so.
215
216
The remaining fields should be fairly self-evident: there's the exec count
217
progress indicator for the current stage, a global exec counter, and a
218
benchmark for the current program execution speed. This may fluctuate from
219
one test case to another, but the benchmark should be ideally over 500 execs/sec
220
most of the time - and if it stays below 100, the job will probably take very
221
long.
222
223
The fuzzer will explicitly warn you about slow targets, too. If this happens,
224
see the perf_tips.txt file included with the fuzzer for ideas on how to speed
225
things up.
226
227
6) Findings in depth
228
--------------------
229
230
  +--------------------------------------+
231
  | favored paths : 879 (41.96%)         |
232
  |  new edges on : 423 (20.19%)         |
233
  | total crashes : 0 (0 unique)         |
234
  |  total tmouts : 24 (19 unique)       |
235
  +--------------------------------------+
236
237
This gives you several metrics that are of interest mostly to complete nerds.
238
The section includes the number of paths that the fuzzer likes the most based
239
on a minimization algorithm baked into the code (these will get considerably
240
more air time), and the number of test cases that actually resulted in better
241
edge coverage (versus just pushing the branch hit counters up). There are also
242
additional, more detailed counters for crashes and timeouts.
243
244
Note that the timeout counter is somewhat different from the hang counter; this
245
one includes all test cases that exceeded the timeout, even if they did not
246
exceed it by a margin sufficient to be classified as hangs.
247
248
7) Fuzzing strategy yields
249
--------------------------
250
251
  +-----------------------------------------------------+
252
  |   bit flips : 57/289k, 18/289k, 18/288k             |
253
  |  byte flips : 0/36.2k, 4/35.7k, 7/34.6k             |
254
  | arithmetics : 53/2.54M, 0/537k, 0/55.2k             |
255
  |  known ints : 8/322k, 12/1.32M, 10/1.70M            |
256
  |  dictionary : 9/52k, 1/53k, 1/24k                   |
257
  |       havoc : 1903/20.0M, 0/0                       |
258
  |        trim : 20.31%/9201, 17.05%                   |
259
  +-----------------------------------------------------+
260
261
This is just another nerd-targeted section keeping track of how many paths we
262
have netted, in proportion to the number of execs attempted, for each of the
263
fuzzing strategies discussed earlier on. This serves to convincingly validate
264
assumptions about the usefulness of the various approaches taken by afl-fuzz.
265
266
The trim strategy stats in this section are a bit different than the rest.
267
The first number in this line shows the ratio of bytes removed from the input
268
files; the second one corresponds to the number of execs needed to achieve this
269
goal. Finally, the third number shows the proportion of bytes that, although
270
not possible to remove, were deemed to have no effect and were excluded from
271
some of the more expensive deterministic fuzzing steps.
272
273
8) Path geometry
274
----------------
275
276
  +---------------------+
277
  |    levels : 5       |
278
  |   pending : 1570    |
279
  |  pend fav : 583     |
280
  | own finds : 0       |
281
  |  imported : 0       |
282
  | stability : 100.00% |
283
  +---------------------+
284
285
The first field in this section tracks the path depth reached through the
286
guided fuzzing process. In essence: the initial test cases supplied by the
287
user are considered "level 1". The test cases that can be derived from that
288
through traditional fuzzing are considered "level 2"; the ones derived by
289
using these as inputs to subsequent fuzzing rounds are "level 3"; and so forth.
290
The maximum depth is therefore a rough proxy for how much value you're getting
291
out of the instrumentation-guided approach taken by afl-fuzz.
292
293
The next field shows you the number of inputs that have not gone through any
294
fuzzing yet. The same stat is also given for "favored" entries that the fuzzer
295
really wants to get to in this queue cycle (the non-favored entries may have to
296
wait a couple of cycles to get their chance).
297
298
Next, we have the number of new paths found during this fuzzing section and
299
imported from other fuzzer instances when doing parallelized fuzzing; and the
300
extent to which identical inputs appear to sometimes produce variable behavior
301
in the tested binary.
302
303
That last bit is actually fairly interesting: it measures the consistency of
304
observed traces. If a program always behaves the same for the same input data,
305
it will earn a score of 100%. When the value is lower but still shown in purple,
306
the fuzzing process is unlikely to be negatively affected. If it goes into red,
307
you may be in trouble, since AFL will have difficulty discerning between
308
meaningful and "phantom" effects of tweaking the input file.
309
310
Now, most targets will just get a 100% score, but when you see lower figures,
311
there are several things to look at:
312
313
  - The use of uninitialized memory in conjunction with some intrinsic sources
314
    of entropy in the tested binary. Harmless to AFL, but could be indicative
315
    of a security bug.
316
317
  - Attempts to manipulate persistent resources, such as left over temporary
318
    files or shared memory objects. This is usually harmless, but you may want
319
    to double-check to make sure the program isn't bailing out prematurely.
320
    Running out of disk space, SHM handles, or other global resources can
321
    trigger this, too.
322
323
  - Hitting some functionality that is actually designed to behave randomly.
324
    Generally harmless. For example, when fuzzing sqlite, an input like
325
    'select random();' will trigger a variable execution path.
326
327
  - Multiple threads executing at once in semi-random order. This is harmless
328
    when the 'stability' metric stays over 90% or so, but can become an issue
329
    if not. Here's what to try:
330
331
    - Use afl-clang-fast from llvm_mode/ - it uses a thread-local tracking
332
      model that is less prone to concurrency issues,
333
334
    - See if the target can be compiled or run without threads. Common
335
      ./configure options include --without-threads, --disable-pthreads, or
336
      --disable-openmp.
337
338
    - Replace pthreads with GNU Pth (https://www.gnu.org/software/pth/), which
339
      allows you to use a deterministic scheduler.
340
341
  - In persistent mode, minor drops in the "stability" metric can be normal,
342
    because not all the code behaves identically when re-entered; but major
343
    dips may signify that the code within __AFL_LOOP() is not behaving
344
    correctly on subsequent iterations (e.g., due to incomplete clean-up or
345
    reinitialization of the state) and that most of the fuzzing effort goes
346
    to waste.
347
348
The paths where variable behavior is detected are marked with a matching entry
349
in the <out_dir>/queue/.state/variable_behavior/ directory, so you can look
350
them up easily.
351
352
9) CPU load
353
-----------
354
355
  [cpu: 25%]
356
357
This tiny widget shows the apparent CPU utilization on the local system. It is
358
calculated by taking the number of processes in the "runnable" state, and then
359
comparing it to the number of logical cores on the system.
360
361
If the value is shown in green, you are using fewer CPU cores than available on
362
your system and can probably parallelize to improve performance; for tips on
363
how to do that, see parallel_fuzzing.txt.
364
365
If the value is shown in red, your CPU is *possibly* oversubscribed, and
366
running additional fuzzers may not give you any benefits.
367
368
Of course, this benchmark is very simplistic; it tells you how many processes
369
are ready to run, but not how resource-hungry they may be. It also doesn't
370
distinguish between physical cores, logical cores, and virtualized CPUs; the
371
performance characteristics of each of these will differ quite a bit.
372
373
If you want a more accurate measurement, you can run the afl-gotcpu utility
374
from the command line.
375
376
10) Addendum: status and plot files
377
-----------------------------------
378
379
For unattended operation, some of the key status screen information can be also
380
found in a machine-readable format in the fuzzer_stats file in the output
381
directory. This includes:
382
383
  - start_time     - unix time indicating the start time of afl-fuzz
384
  - last_update    - unix time corresponding to the last update of this file
385
  - fuzzer_pid     - PID of the fuzzer process
386
  - cycles_done    - queue cycles completed so far
387
  - execs_done     - number of execve() calls attempted
388
  - execs_per_sec  - current number of execs per second
389
  - paths_total    - total number of entries in the queue
390
  - paths_found    - number of entries discovered through local fuzzing
391
  - paths_imported - number of entries imported from other instances
392
  - max_depth      - number of levels in the generated data set
393
  - cur_path       - currently processed entry number
394
  - pending_favs   - number of favored entries still waiting to be fuzzed
395
  - pending_total  - number of all entries waiting to be fuzzed
396
  - stability      - percentage of bitmap bytes that behave consistently
397
  - variable_paths - number of test cases showing variable behavior
398
  - unique_crashes - number of unique crashes recorded
399
  - unique_hangs   - number of unique hangs encountered
400
  - command_line   - full command line used for the fuzzing session
401
  - slowest_exec_ms- real time of the slowest execution in ms
402
  - peak_rss_mb    - max rss usage reached during fuzzing in mb
403
404
Most of these map directly to the UI elements discussed earlier on.
405
406
On top of that, you can also find an entry called 'plot_data', containing a
407
plottable history for most of these fields. If you have gnuplot installed, you
408
can turn this into a nice progress report with the included 'afl-plot' tool.